Asymptotic Average Redundancy of Huffman (and Shannon-Fano) Block Codes

We study the redundancy of Huffman code (which, incidentally, is as old as the author of this paper). It has been known from the inception of this code that in the worst case Huffman code redundancy defined as the excess of the code length over the optimal (ideal) code length is not more than onc. Over more than forty years, insightful, elegant and useful constructions have been set up to determine tighter bounds on the redundancy. One should mention here Gallager's upper bound of PI +0.086 (PI is the probability of the most likely symbol), and tighter bounds due to Capacelli, de Santis and others. However, to the best of our knowledge no precise asymptotic results have been reported in literature thus far. We consider here a memoryless binary source generating a sequence of length n distributed as binomial(n,p) with p being the probability of emitting O. Based on recent result of Stubley, we prove that for p ¥1/2 the average redundancy Rn of the Huffman code as n -)00 becomes { ~ lJ~2 = 0.057304 ... R" ...... , '(aM) 'l 1 2-{n{JM)/M 2" M i1 n 2" M(l 2 l)M) Ct = log2(1 p)/p irrational Ct = Z rational where M, N are integers such that gcd(N, M) = 1, (x) = x LxJ is the fractional part of x, and [3 = -log2(1 pl. The appearance of the fractal~like function ([3Mn) explains the erratic behavior of the Huffman redundancy, and its "resistance" to succumb to a precise analysis. In fact, from the above we also can recover the Gallager's upper bound. As a side result, we prove that the average redundancy of the Shannon-Fano code is { . , R" ...... ~ 11 ((Mn[3) t) Ct = !og2(1 p)/p irrational Ct = Z rational as n -)00. These findings are obtained through analytic methods such as Fourier analysis and theory of distribution of sequences modulo 1. Index Terms Huffman code, Shannon-Fano code, average redundancy, Fourier analysis, distribution of sequences, Weyl's criterion. "This research was supported in part by NSF Grants NCR-9415491 and C-CR-9804760.